1. Field
This application relates generally to computer graphics, and more specifically to computer systems and processes for rendering scenes using deep-framebuffer rendering techniques.
2. Related Art
Images used in high-quality motion pictures need to be photorealistic. Since the mid-1980s, algorithms utilizing the Reyes rendering architecture have been used for fast, high-quality rendering of complex scenes of motion pictures. The Reyes rendering architecture was developed in the mid-1980s with the specific goal of creating a rendering algorithm that would be applicable to creating special effects for motion pictures. The architecture efficiently achieves several important effects: smooth curved surfaces, surface texturing, motion blur, and depth of field.
However, the Reyes rendering architecture generally requires the entire framebuffer (a buffer that stores the contents of an image pixel by pixel) to be stored in memory since the final image cannot be outputted until all geometric primitives have been processed. Bucket rendering is a memory optimization technique in which the framebuffer is subdivided into buckets. Objects in a scene are placed into the buckets based on their location within the scene. Each bucket is rendered independently, and the data from a previously rendered bucket is discarded before the next bucket is processed. As a result, only the portion of the framebuffer corresponding to the current bucket and the high-level descriptions of all geometric primitives must be maintained in memory. The benefits of bucket rendering include a decrease in the size of the working set of framebuffer memory required during rendering and a reduction of the amount of Z-buffer memory (the buffer for storing the depth of each pixel) required to render high-resolution images with high-depth complexity, motion blur, and/or depth of field. With bucket rendering, the shading step may be deferred until visibility determination is completed within the bucket being rendered, and unnecessary shading of hidden surfaces may be eliminated.
However, using bucket rendering in a Reyes rendering architecture has a number of drawbacks. One drawback is the redundant computations required in processing geometric primitives when the full list of primitives does not fit entirely in memory and when primitives span multiple buckets, as is the case in most complex scenes. Bucket rendering requires careful memory management to handle such complex scenes. In some implementations, the primitive data from a previously rendered bucket is discarded before the next bucket is processed. As a result, re-processing/regeneration of the same geometric primitives that span subsequent buckets results in a significant amount of redundant computation. In some implementations, the geometric primitives spanning multiple buckets may be kept in memory until all the subsequent spanned buckets are rendered. However, when the list of accumulated geometric primitives has filled up the memory devoted to that purpose, some of these geometric primitives are discarded anyways and re-processing/regeneration of these geometric primitives still results in expensive redundant computations. The geometry shaders responsible for generating geometry may be coded to only focus on primitives that span the bucket(s) being rendered so as to reduce the amount of redundant computation associated with bucket rendering. However, the geometry shaders become much more difficult to write.
As an alternative to Reyes rendering, deep-framebuffer rendering algorithms may be employed for fast, high-quality rendering of complex scenes of motion pictures. Unlike a traditional framebuffer, which only stores color, opacity, and depth information per pixel, a deep-framebuffer caches the information of pieces of geometry visible on each pixel of an image from the point of view of a virtual camera. During the rasterization process, pieces of geometry that are fully occluded by other opaque pieces of geometry are removed from the deep frame-buffer. Therefore, the deep-framebuffer only stores pieces of geometry that are visible directly from the camera or visible through other semi-transparent and non-refractive pieces of geometry. Besides defining which pieces of geometry are visible on each pixel of an image, the deep-framebuffer also defines how much the pieces of geometry overlap when viewed from the virtual camera. This extra information stored in the deep-framebuffer, but not in a traditional framebuffer, is useful for evaluating shading later down the rendering pipeline. A scene may be shaded multiple times after adjusting shading parameters without repeating any geometry generation, tessellation, or rasterization steps because the result of these steps is already stored in the deep-framebuffer.
Typically, the information stored in the deep-framebuffer includes a z-sorted list (i.e., sorted by depth from the camera) of visibility masks for each pixel. For example, the visibility mask may be an A-buffer coverage mask. Each visibility mask is a bit mask with each bit representing a particular sub-pixel. Each visibility mask defines which portion of that pixel is covered by a specific piece of geometry; the visibility mask may reference the corresponding geometry data structure, which stores per primitive data and/or per vertex information of the geometry, including the vertices' position, (u, v) texture coordinates, and position derivative vectors with respect to each of the (u, v) texture coordinates. This set of information is sufficient to evaluate shading further down the rendering pipeline.
In some examples, the pieces of geometry as described above are micropolygons. A micropolygon is a quadrilateral or a triangle that is about the size of a pixel when projected onto a screen. At rendering time, a geometric primitive—e.g., parametric surfaces (such as non-uniform rational B-spline, or NURBS, surfaces), subdivision surfaces, parametric curves, or polygons (large triangles or quads)—may be tessellated into micropolygons. Using micropolygons, a highly detailed image may be obtained. Displacement mapping may be applied to micropolygons before they are rasterized. Displacement mapping is a technique where an image texture or a procedural texture or a height map is used to cause an effect where the micropolygon vertices of the textured surface are displaced, often along the local surface normal, according to the value the texture function evaluates to at each point on the surface. It gives surfaces a great sense of depth and detail, permitting in particular self-occlusion, self-shadowing, and complex silhouettes.
It should be recognized that a micropolygon may span multiple pixels, because micropolygons may not be perfectly aligned with pixels. Motion blur and depth of field are other factors that may result in the micropolygon-to-pixel relationship being one-to-many. Therefore, a geometry data structure corresponding to a micropolygon may be referenced by several visibility masks, one for each spanned pixel.
The deep-framebuffer may be saved into a single file (a deep file) and it may be read back from the deep file into memory for further processing. However, saving the deep-framebuffer into a single file at the end of a rendering process may limit the complexity or the richness of the scenes and the productivity of the artists who are generating the scenes. As scenes become more complex, the deep-framebuffer may grow well beyond the amount of memory available on the processing machine. A rendering process that has been running for hours or days may be terminated by the system because the size of the deep-framebuffer exceeds the memory available, and the whole rendering process must be restarted again after manual adjustments are made by human artists. An artist may be forced to choose between limiting the complexity or richness of the scenes, or implementing tedious workarounds, such as manual level-of-detail (LOD) techniques (i.e., techniques that decrease the complexity of 3D object representation based on object importance, eye-space speed, or position) when overly complex or large scenes must be generated by the rendering tools.
As an alternative to saving the deep-framebuffer into a single file at the end of a rendering process, memory optimization techniques may be used to subdivide a deep-framebuffer into tiles. Similar to bucket rendering techniques used in the Reyes rendering architecture, tiling techniques subdivide a scene and the deep-framebuffer corresponding to different parts of the scene into non-overlapping tiles. An object in a scene may span multiple tiles and portions of the object may be placed into each of those tiles. Rasterization is performed one tile at a time. Each tile is rendered independently, and the data from a previously rendered tile is discarded before the next tile is processed. As a result, only the portion of the deep-framebuffer corresponding to the current tile is maintained in memory. Thus, the size of the working set of deep-framebuffer memory required during rendering may be reduced.
However, similar to bucket rendering, one significant drawback of tiling is the redundant computations required in processing geometric primitives when the primitives span multiple tiles. Because the data from a previously rendered tile is discarded before the next tile is processed, re-processing of the same geometric primitives for subsequent tiles results in redundant computations. These redundant computations may be substantial, especially for geometries with long geometry generation, tessellation, and rasterization times or geometries that appear extensively over a scene, such as fur, hair, grass, trees, foliage, and the like.